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Abstract 

Active galactic nuclei (AGN) have been suggested to be sources of very high 
energy neutrinos. We consider the possibility of using AGN neutrinos to test 
neutrino mixings. From the atmospheric, solar and laboratory data on neu- 
trino oscillations we derive the flavour composition of the AGN neutrino flux 
in different neutrino mixing schemes. We show that most of the schemes con- 
sidered can be distinguished from each other and the existence of a sterile 
neutrino can be specially tested. AGN neutrinos can also be used to test 
those four-neutrino scenarios where solar neutrinos oscillate into an arbitrary 
mixture of Vs and Vr- 
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Introduction. Atmospheric and solar neutrino measurements have provided a strong 
evidence for the existence of neutrino oscillations, or neutrino masses and mixings. 
The recent atmospheric neutrino results from Super-Kamiokande [|l| suggest the 
oscillation of the muon neutrino (z/^) into a tau neutrino (z/^) or a sterile neutrino 
(i/g). On the other hand, the observed solar neutrino deficit can be interpreted as 
evidence for the oscillation of the electron neutrinos (ug) into neutrinos of a different 
flavour (see e.g. ^). In addition, the LSND collaboration has reported results of 
their laboratory measurements that indicate the existence of z/^ — > z/g and z/^ — > z/g 
oscillations Unfortunately, the data does not yet uniquely fix the mass and 
mixing pattern among neutrinos. A number of different scenarios are still allowed. 

The explanation of the solar, atmospheric and LSND results requires a four- 
neutrino scheme of three active neutrinos z/g, I'fi, Vt and one sterile neutrino Vs 0- 
The mass pattern should have a two-doublet structure |0: one of the doublets 
consists of Ve and Vg (or and z/^), and is responsible for the solar neutrino deficit, 
and the other one, responsible for the atmospheric neutrino anomaly, consists of z/^ 
and Vr (or z/^ and z/^). These doublets are separated by a wide mass gap (C(l eV)), 
so as to explain the LSND result in terms of z/g — mixing. 

There is an ambiguity in this scheme due to the fact that there are at present four 
viable interpretations of the solar neutrino data corresponding to different choices 
of the oscillation parameters: the vacuum oscillation solution (VO), the low MSW 
solution (LOW), the small mixing angle MSW solution (SMA) and the large mixing 
angle MSW solution (LMA) p|, ||. Disregarding the LSND data, it is, of course, 
possible to give an explanation for the other results in terms of the known three 
neutrino species. In this case one assumes that the atmospheric neutrino anomaly 
is due to z/^ — Vr mixing and solar neutrinos oscillate to z/^ and z/^. On the other 
hand, there might exist two (or more) sterile neutrinos (z^s, ^s') so that the solar 
neutrino deficit is due to z/g — mixing and the atmospheric neutrino anomaly due 
to — Vg' niixing. 

In this paper we will study the possibility of testing the various neutrino mixing 
scenarios by observing the flavour composition of the high-energy neutrino flux from 
active galactic nuclei (AGN). With the oscillation parameters suggested by the solar 
neutrino, the atmospheric neutrino and the laboratory measurements, the fluxes of 
different neutrino flavours are determined entirely by the mixing angles with no 
dependence on the masses. It turns out that there are quite large variations of 
the fluxes between the different mixing schemes, which should be detectable in the 
new neutrino telescopes such as AMANDA, NESTOR, BAIKAL, ANTARES 
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and NEMO. With sufficient statistics, which we beheve to be achievable in these 
telescopes within a reasonable time scale, one would be able to discriminate between 
different solar neutrino solutions, check the possible existence of large active-sterile 
mixings, and make distinction between three-neutrino and most of the four-neutrino 
schemes. 

Production of AGN neutrinos. The AGN neutrino production is suggested to take 
place in the AGN cores §], in the jets and at the endpoints of jets so-called 
hot spots flOl- In these sources, charged particles are supposed to be accelerated 



in the vicinity of shock waves by the ffist order Fermi acceleration mechanism (see 
e.g. [jlll). The collisions of the accelerated protons with photons in the ambient 



electromagnetic fields would then lead to the production of energetic gamma rays 
and neutrinos via the pion photoproduction processes |T^, 



PI ^ ^ 27. (1) 
The neutrino and high energy photon spectra are thus related to each other, and 



the neutrino flux can be estimated from the high energy photon flux [12, 13|. Tau 



neutrinos are produced in negligible amounts . Hence the flavour composition of 



the AGN neutrino flux is expected to be, with a high precision, 

Nr:N^:Ne = 0:2:1, (2) 

where Na, a = e, fi, r, denotes the total flux of z/q,'s and z/^'s. 

The integrated diffuse intensity of high-energy {Ei, > 10 TeV) neutrinos from 
AGN and blazars detected in a detector with an effective area of 1 km^ has been 
estimated to be a few hundreds per year |jl3|. When the plans to build several 



km-scale detectors become reality, the total annual flux observed will be, if these 
estimates turn out to be correct, of the order of some hundreds to one thousand 
neutrinos. This is much larger than the estimated background flux of atmospheric 
neutrinos in the same energy range. On the other hand, the flavour composition of 
the atmospheric neutrino flux will be quite well understood due to the measurements 
at the lower part of the energy spectrum. Hence the background of the atmospheric 
neutrinos would not crucially interfere with the determination of the AGN neutrino 
fluxes. Therefore the detection of AGN neutrinos will probe new physics beyond 
the Standard Model (SM), in particular neutrino oscillations []T4| . 

It is important to notice that all three neutrino flavours will be observed inde- 
pendently from each other [l^, 0, |l^. Therefore any new physics beyond the SM 
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that may alter the flavour composition of the neutrino flux will be probed. Neutrino 



oscillations can be tested by using the flavour composition |]T^, but AGN neutrino 
flux is sensitive also to other phenomena, such as new interactions, magnetic mo- 
ments and decays of neutrinos, which may affect the intensity or the composition of 
the flux |]I6|. Here we shall study how the AGN neutrino data might be used to dis- 
criminate between various neutrino mixing schemes proposed for the interpretation 
of the results of the solar, atmospheric and LSND neutrino experiments. 

Oscillation probabilities. In the four-neutrino scenarios mentioned above, the neu- 
trinos form two pairs of flavours with a large or potentially large mixing within each 
pair. In the two basic scenarios that can be considered p all the other mixing angles 
are small and they cannot substantially affect the flavour composition of the AGN 
neutrino flux. In that case it is sufficient to consider two-flavour oscillations with 
either atmospheric or solar neutrino mixing angles ^atm and ^0, respectively. 

The oscillation probability for the two-neutrino oscillation z/q, —>■ {a (3) is 
given by 

1 / f lS.m?L\\ 

Pii^a ^^p) = 2 ^i^' 2^"/^ [I - COS [-^J J , (3) 

where Am^ is the squared mass difference of the corresponding mass eigenstates and 
9ai3 is the mixing angle. Due to the enormous distance L to the source (typically 
L ~ 500 Mpc) the AGN neutrino oscillations can be relevant to probe squared 
mass differences of the order 10~^^ eV^ for the typical AGN neutrino energies of the 
order of PeV. Turning this argument around, one can say that for any value of Am^ 
implied by the present experimental results the oscillating term averages to zero 
when integrated over the neutrino energy spectrum. Obviously, matter effects could 
only be relevant for the neutrinos that pass across the Earth. But, in this case, for 
such high energies, the weak eigenstates form effective mass eigenstates and decouple 
from each others. The transition probabilities P(z/q — z/^) and P(z/q Va) are thus 
entirely determined by the mixing angles Oap between the flavour states Va and vp: 

Pii^a ^iyp) = ^ sin^ 2e^p, P(z/, z/,) = 1 - 1 sin^ 26^^ . (4) 

Four-neutrino models. Let us first consider the four-neutrino scheme where z/^ mixes 
maximally or almost maximally (0.8 < sin^ 26'atm < 1-0) with u^- to account for the 
atmospheric neutrino anomaly while the oscillations of z/g into sterile neutrinos z/^ 
explain the solar neutrino deficit. Let = 1, = 2, Nr = he the normalized 
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neutrino fluxes with the relative rates as predicted by the SM, that is, if there are no 
neutrino mixings. Then, the predicted rates for z/g, u^, and Vs in this four-neutrino 
scenario are 

1 
2 



A^, = 1 - ^ sin^ 2^o , 



iV^ = 2-sin2 2^atm, 
Nr = sin^ 29^tm , 

iV, = isin2 2^0. (5) 

According to recent studies only a small MSW mixing angle 6q can fit the solar 
neutrino data with a sterile neutrino option p|, Q . Thus the z/g — niixing will not 
substantially affect the composition of the AGN neutrino flux, and one predicts the 
following observable ratios of neutrino fluxes: 

2Ne 



2N^ 



= sin^ 26'atm . (6) 

The latter ratio would provide a new independent measurement for the atmospheric 
neutrino mixing angle 6'atm- 

In the other four-neutrino scenario, the atmospheric neutrino anomaly is ex- 
plained in terms of — Ug oscillations and the solar neutrino deficit in terms of 
— Uj. oscillations. In the solar case various types of solutions can fit the data, 
three of them with a large mixing angle P, ^: the vacuum oscillation solution (VO) 
with 0.7 < sin2 2^0 < 1.0, the LOW solution with 0.9 < sin2 2^0 < 1.0 and the 
large mixing angle (LMA) MSW solution with 0.55 < sin^ 29 < 0.96. The latter is 



now preferred as a result of the Super-Kamiokande energy spectrum data [0. In 
all these three cases, sin^2^Q lies above 1/2, and this is the constraint we will use 
below. In the small mixing angle (SMA) MSW solution sin^ 29q is very small and 
can be neglected. The u^. and the z/^ rates are the same as in the previous scenario, 
given in (||), but the u^. and the z/^ rates are interchanged, i.e. 

= ^sm^2eQ , 

AT, = sin2 2^atm. (7) 
One interesting observable flux ratio is 

^=-'2<)e, (8) 
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which provides an independent measurement of the solar neutrino mixing angle 6q. 
Naturally this ratio distinguishes between the large solar mixing angle solutions, i.e. 
VO, LMA and LOW, and the SMA solution: there is no v^- component in the flux 
in the latter case. Another useful observable is the ratio of the flux to the Vf. and 
z/t- fluxes, for which one has the bounds 

1 ^ ^^^^ = 2 - sin^ 2^,tm < 1.2 , (9) 

where the upper limit comes from the constraint sin^26'atm ^ 0.8. (Note that if 
neutrinos did not mix this ratio had the value 2.) 

The ratio can be used to make a distinction between the scheme where the 
solar neutrino deficit is explained with the — Vt oscillations and the scheme where 
it is done with a z/g — z/^ mixings. In the latter case (see Eqs. (|^)) the ratio is 
bounded as 

- < ^ — < - . 10 

The contrast between the two four- neutrino scenarios is evident. Let us note that if 
the solar neutrino deficit is due to large z/g — mixing (sin^26'0 > i), the ratio (H) 
varies between 0.7 and 0.9. This solution is, however, strongly disfavoured by the 
present data 0, ^. 

The predictions of the different neutrino mixing scenarios can be described in 
terms of the relative 'abundances' of neutrino flavours defined as 

Va = ir; r; a = e,u,r, (11) 

which obey the relation ye + Vfi + Vt = 3. In the case of no mixing (SM) one has 
He = I, = 2 and y-r = 0. In Fig. 1 we present the allowed regions in the — yr 
plane for each of the scenarios. The two straight solid lines correspond to SMA 
MSW solutions of the solar neutrino deficit and the two quadrangular areas with 
dashed lines to all of the solutions with a large mixing angle, 0.5 < sin^ 29q < 1 
(LMA, LOW and VO). The straight line labeled with z/^ corresponds to the four- 
neutrino scheme where — Ug mixing solves the solar neutrino deficit, and the other 
line (labeled with z/^-), lying on the ?/^-axis, corresponds to the scenario where it is 
explained in terms of — Ur mixing. The lower quadrangular area labeled with Ur 
corresponds to the scenario where z/g — mixing solves the solar neutrino deficit 
and the area labeled with to the scheme where it is solved by z/g — z/^ mixing and 
the atmospheric neutrino anomaly by z/^ — z/,- oscillations. As mentioned above, the 
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latter is disfavoured by the solar neutrino data P, |^. Along each of the solid or 
dashed lines either 6q or 6'atm is constant. As sin^ 26'atm increases from 0.8 to 1 the 
abundance always decreases, and the growth of sin^ 20q from 0.5 to 1 is indicated 
by arrows. Hence the maximal mixing point sin^ 26q = sin^ 26'atm = 1 is the upper 
left corner in the lower "quadrangle" and the upper right corner in the other one. 

The basic scenarios studied so far may be considered as extreme cases of a more 
general four-neutrino scenario where solar electron neutrinos oscillate not to a pure 
sterile or a pure active u^- neutrino flavour but rather to some linear combination 
Ux = cos 6 Us — sinO Vr of those two. The atmospheric muon neutrinos would then 
oscillate into the corresponding orthogonal superposition Vy = sin 6 Vg + cos 6^ v^-. 
This kind of situation is perfectly consistent with the limits derived from the 
reactor disappearance experiments and with the atmospheric neutrino observations 
as well. As to solar neutrinos, so far there does not exist a data analysis for this 
Ve — Vs — three-neutrino mixing scheme, as complete as what has been done for 



the — Vg or — two- neutrino oscillation schemes 0, |T3- Let z/i, V2 be 
the mass eigenstates responsible for the solar neutrino deficit and z/3, z/4 the states 
involved in atmospheric neutrino oscillations. The two pairs are separated by the 
LSND mass gap but no other specific mass hierarchy has to be assumed. The reactor 
experiments constrain the mixing matrix elements f/gS; f^e4; U^i, U^2 to be small ^ 
but not Uri or Usi- If one drops all the mixing angles that are necessarily small 
(unimportant for our purpose), the mixing matrix is given as follows: 

ui = cos 6*0 z/g — sin 6*0 (cos 6^ — sin 9 u^) , 
U2 = sin 6*0 Ue + cos 6^0 (cos 6* z/^ — sin6' Ur) , 

z/3 = cos 6'atm ^,1 - sin 6'atm (sin 6^ + COS 6 Ur) , 

z/4 = sin 6'atm + COS 6'atm(sin 9 Us + COS 9 z/^) . (12) 

It is clear that the solar and atmospheric neutrino phenomena are understood in 
terms of z/g — Ux and z/^ — Uy oscillations, respectively (z/^, and Uy are the states inside 
brackets) . 

The predicted rates of AGN neutrinos are now 

Nr = cos^ 9 sin^ 26'atm + ^ sin^ 9 sin^ 29q , 

Ns = sin^ 9 sin^ 29. + ^ cos^ 9 sin^ 29q , (13) 

while the rates of z/g and z/^ remain the same as in Eqs. (|^). If the angle 9 is set to 
be an arbitrary parameter the allowed regions are enlarged covering the two more 
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specific schemes considered before. Tliere are two different regions, botli limited by 
dotted lines: one is for solar SMA solutions and the other for large solar mixing 
angle solutions LMA, LOW and VO. The angle 9 varies along the dotted lines: one 
has ^ = for the z/g — Vg solar neutrino mixing scheme and 6* = 7r/2 for the z/g — Vj. 
mixing scheme. If the AGN neutrino data turns out to lie outside the areas for 
these two specific schemes it will indicate that solar neutrinos do not oscillate to 
pure active or sterile neutrino states but rather to a mixture of them. 

Three-neutrino model. A radically different scenario is the one with just the three 
standard model neutrinos. That is enough for explaining the atmospheric and solar 
neutrino data when the LSND results are disregarded. It is worthwhile to check if the 
AGN high-energy neutrinos can be used to confirm or disprove such a three-neutrino 
hypothesis. In the viable three-neutrino scenario the atmospheric i/^'s essentially 
oscillate to but not to z/g, as indicated by the Super-Kamiokande and Chooz 
data |T^. If and Vr are maximally mixed the prediction for the AGN neutrinos 
is that half of the z/^'s are converted into z/^-'s making all the observable fluxes of 
the flavours z/g, z/^ and Vr equal to each other, irrespectively to the value of the 
solar neutrino mixing angle 9q. The situation changes, however, if the atmospheric 
neutrino mixing is not maximal. 

In the three-neutrino scheme the solar electron neutrinos oscillate into a par- 
ticular superposition of z/^ and with which z/g forms a pair of almost degenerate 
mass eigenstates, z/i and z/2, separated from the mass eigenstate z/3 by the mass gap 
Am^tm — (2 — 6) X 10^'^ eV^. The limits obtained by the Super-Kamiokande and 
Chooz |18[ experiments imply that z/g is only a small fraction of the mass eigenstate 



z/3, that is, the element f/g3 of the rotation matrix is small. For our purpose this 
mixing can be neglected and will be taken to be zero. Then the matrix elements Uai 
can be read from 

z/g = cos 6q ui + sin 6q z/2 , 

= cos 6'atm(- siu 9q Ui + cos 9qU2) + sin 6'atm 1^3 , 

= - sin 6'atm(- sin 9q vi + cos 9q z/2) + cos 9^^„, z/3 . (14) 

The mass eigenstates z/i and z/2 are of course separated by the gap Am|. 

After many oscillation lengths the number of neutrinos with a given flavour, N^., 
averages to an amount that is related to the rates of originally emitted neutrinos. 
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Ng, as follows: 



2 



N^ = J2\U^.UI iV°. (15) 

i,f3 



Using the same normalization as before, i.e. = 1, = 2, N!^ = 0, one obtains 



Ne = 1 + ^sin22^0cos2^atm , 

= 2- Sin^ 26'atm - ^ Sin^ 26'© cos 26'atm COS^ 6'atm , 

A^^ = sin^ 26'atm - ^ sin^ 26'0 cos 26'atm sin^ 6'atm • (16) 

The total number of neutrinos is conserved and equal to 3 in this normalization. If 
6'atm is maximal one gets equally distributed fluxes over all flavours, as expected. If 
6'atm is not maximal the relative rates change and, interestingly enough, are sensitive 
to the sign of cos26'atm5 thus differentiating between a mixing angle 6'atm larger than 
7r/4 from a mixing angle smaller than tc/A. This can be seen in Fig. 1, where the two 
small triangular-like areas correspond to sin^ 26'atm between 0.8 and 1 and sin^ 26'0 
from 1/2 to 1. The upper "triangle" corresponds to a negative cos 26'atm and the 
lower "triangle" to a positive cos2^atm; and they meet at the point of maximal 
mixing, ^atm = vr/4. Hence, the AGN neutrinos may be used, in principle at least, 
to discriminate between a negative and a positive cos 26'atm- The sensitivity is best 
when sin^ 26'atm = 0.8 and sin^ 26'0 = 1, corresponding to the most distant points 
of the two "triangles". The small 6q solution (SMA) identifies with the straight 
line starting at the (1, 1) point, the same as for the four- neutrino scheme with solar 
SMA — mixing. Apart from that, the three-neutrino scenario is in general quite 
distinct from the four-neutrino schemes. 

Exotic scenarios. One can find in the literature still some other oscillation scenarios 
in addition to the three-neutrino and four-neutrino models considered above, which 
may lead to different predictions for the flux composition. Models have been pro- 
posed [1^ with more than one sterile neutrino where the solar neutrino deficit is 
explained in terms of z/g — Vs mixing and the atmospheric neutrino anomaly in terms 
of z/^ — Vs' mixing, where z/^ and Vgi are two separate sterile states. The prediction 
is in that case, within the SMA solar neutrino solution, 1.5 < 1//^ < 1.6 and yr = 0, 
the same as in the four-neutrino scheme with solar — Vr SMA solution. These two 
cases cannot be distinguished with AGN neutrino data only. Of course, there could 
also exist a sterile neutrino very degenarate with one of the active neutrinos, so that 



8 



the squared mass difference is below the values probed in other phenomena. Such a 
neutrino could have visible effects on the AGN neutrino ffux if its mixing with one 
of the active neutrinos is large. 

Cosmic connection. Let us finally note that the question of existence of sterile 
neutrinos and their mixings with active neutrinos has relevance also for cosmology. 
Depending on the value of Am^, a large active-sterile neutrino mixing could bring 
sterile neutrinos into thermal equilibrium thereby increasing the effective number of 
light neutrinos, N^, . According to a recent analysis ||21| the abundancies of light 
elements yield the upper bound < 3.20, which sets very tight constraints on the 
active-sterile mixings, in particular on the z/^ — z/^ mixing [^. The four-neutrino 
scenario where the atmospheric neutrino anomaly is explained in terms of the — Ug 
mixing seems hence unprobable. A large Ve — i-'s mixing is less stringently constained 
by this argument as the value of Am^ for the solar neutrinos is smaller than for the 
atmospheric neutrinos. The cosmological bound on Ny is a disputable matter. Some 
observations have indicated a higher primordial deuterium abundancy than normally 
assumed, and this leads to a less restrictive limit on the effective number of light 
neutrinos [Q. The conflict with the nucleosynthesis bounds may also be avoided 
with a large lepton asymmetry [|^. The AGN flux measurements would provide a 
new way to check the consistency of this cosmological reasoning. 

Conclusions. In conclusion, if neutrinos turn out to be produced in AGN with the 
rates as suggested by some authors [|], ||, [l^, then the measurement of the flavour 
composition of the AGN neutrino flux will provide a new method to discriminate 
between various neutrino oscillation schemes for solar, atmospheric and laboratory 
neutrinos. It would also offer a new independent measurement of the various mixing 
angles, such as the atmospheric and solar neutrino mixing angle 6'atm and 6q. Of 
particular interest is the possibility of testing the existence of sterile neutrinos and of 
making a distinction between small and large angle active-sterile mixing. Moreover, 
one may have a hint of the more general four-neutrino scenarios with large Vr ~ 
mixing, which depart from the basic solar neutrino hypothesis of oscillations into 
pure active or sterile neutrino states. 

The AGN neutrino data will come along with the other present and future solar, 
terrestrial and atmospheric neutrino experiments which will provide a cross-checking 
of neutrino mass and mixing patterns. There are also other possibilities to study 
neutrino oscillations with a long flight distance by observing the neutrino flux from 



the Galactic Center 25 
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Figure 1: Predictions for the rates, Ua-, of AGN z/^, Ur and z/g fluxes at the Earth, 
normahzed as He + Hfi + Hr = 3. The point 'SM' corresponds to the no-mixing case. 
The sohd segment labeled with Ug (z/,-) holds for a scenario with SMA solar z/g — Ug 
(z/g — Ur) oscillations and atmospheric z/^ — Ur {v^ — Vs ) oscillations. The areas 
enclosed by dashed lines are associated with large mixing angle {siv?26Q > 1/2) 
solar neutrino solutions LMA, LOW or VO. The case of — Vs oscillations is labeled 
with Ug and the case of Ve — oscillations is labeled with Vr- The regions limited by 
dotted lines correspond to the four-neutrino scenario where solar neutrinos oscillate 
to an arbitrary mixture of Vs and Vr which is decoupled from its orthogonal mixture 
by the LSND mass gap. The two triangular-like areas are predicted by the LMA 
three- neutrino scenario, with cos26'atm < in the upper "triangle" and cos26'atm > 
in the lower one. In all cases sin^2^atm varies from 0.8 to 1. The arrows indicate 
how sin^ 26 Q grows from 1/2 to 1. 
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